Skip to content

fix(claude-channel): benign dedup for shared-agent reply race#1309

Open
sh1nj1 wants to merge 6 commits into
mainfrom
fix/channel-shared-agent-reply-dedup
Open

fix(claude-channel): benign dedup for shared-agent reply race#1309
sh1nj1 wants to merge 6 commits into
mainfrom
fix/channel-shared-agent-reply-dedup

Conversation

@sh1nj1

@sh1nj1 sh1nj1 commented Jun 16, 2026

Copy link
Copy Markdown
Owner

Problem

Running two claude --channels plugin:collavre@collavre sessions in the same working directory and sending a message in a Collavre topic: both sessions receive the dispatch, the first replies fine, the second fails with:

The reply tool returned 403 Not authorized

Root cause

Same-directory sessions share one Collavre agent (the default AGENT_NAME case). Both subscribe to agent:user:<id>, so a single work-topic dispatch fans out to both and each session's turn calls /reply with the same task_id. This is by design — ClaudeChannelAdapter documents that a work topic "may be handled by any session (the server's atomic task claim dedups concurrent handlers)", with the loser getting a clean 409.

But resolve_reply_agent scoped its task lookup to status: "delegated". Once the winning session's claim_delegated_task flipped the task to done, the loser's lookup returned nil and reply rendered a misleading 403 at the auth layer — one step before the intended 409 dedup. The MCP plugin throws on any non-2xx, so the 403 surfaced to the user as a hard error.

The existing test concurrent replies for same task_id only one wins; loser gets 409 even admitted this in its assertion (weakened to refute 2xx with a comment noting the real result was 403).

Fix

Server (resolve_reply_agent): drop the status: "delegated" filter. Resolve the agent for any task on the topic owned by the caller's Claude Channel agent, and let claim_delegated_task (which keeps the atomic WHERE status='delegated') decide actionability. The loser now reaches the designed 409. Nonexistent / foreign-agent / wrong-topic task_ids still 403 (ownership + topic scoping unchanged).

Plugin (collavre-client.ts + index.ts): reply() treats 409 as a benign dedup (handled: false) instead of throwing; the reply tool reports "already answered by another session" rather than a failed reply.

Verification

  • engines/collavre/test/controllers/api/v1/agents_controller_test.rb — 69 runs, 0 failures (updated the concurrent-reply test to assert :conflict).
  • Plugin: tsc clean build + npm test (28 pass).
  • Rubocop clean; pre-push full suite green.

🤖 Generated with Claude Code

Two Claude Code channel sessions in the same working directory share one
Collavre agent (default AGENT_NAME). Both stream from agent:user:<id>, so a
single work-topic dispatch fans out to both and each session's turn calls
/reply with the same task_id. The server is designed to dedup these via the
atomic task claim (claim_delegated_task), returning 409 to the loser.

But resolve_reply_agent scoped its task lookup to status: "delegated". Once
the winning session flipped the task to "done", the loser's lookup returned
nil and reply rendered a misleading 403 "Not authorized" — which the MCP
plugin threw as a hard error and surfaced to the user.

Server: resolve_reply_agent no longer gates on status. It resolves the agent
for any task on the topic owned by the caller's Claude Channel agent and lets
claim_delegated_task decide actionability — so the loser reaches the intended
409. Nonexistent/foreign task_ids still 403 (ownership + topic scope unchanged).

Plugin: reply() treats 409 as a benign dedup (handled:false) instead of
throwing; the reply tool reports "already answered by another session" rather
than a failed reply.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 90efe44df7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +95 to +97
if (res.status === 409) {
const respBody = await res.text();
return { handled: false, reason: respBody };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not swallow every reply conflict as dedup

When the original dispatch is cancelled or failed instead of answered (for example, offline-session cancellation or stuck-task recovery flips a delegated task while Claude is still composing), /reply now also produces a 409 because the server resolves non-delegated tasks and the atomic claim fails. This branch treats every 409 as a benign sibling-session dedup, and index.ts then reports “Already answered” without posting any comment, so a valid response can be silently dropped. Please distinguish the completed/dedup case from cancelled/failed/not-delegated conflicts before suppressing the error.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 96bb8ac. The claim fails identically in both cases, which is exactly why the client could not tell them apart: claim_delegated_task returns nil for any task that is not delegated, so a dispatch cancelled by an offline session, failed, or flipped by stuck-task recovery while Claude was composing produced the same bare 409 as the sibling-session dedup. The dedup case has an answer already posted; the others have none, and the text in hand is the only copy of it — so index.ts reporting "Already answered" and posting nothing lost a real reply, silently.

The server now names the cause rather than leaving the client to guess:

  • TaskClaimService#conflict_reasonalready_completed when the task is done (a sibling won the claim, or the agent's own job completed it — either way the dispatch was answered and this reply is a duplicate), not_delegated for every other status.
  • reply renders it beside the existing message: { error: ..., reason: ... }, still 409.

The client suppresses on the marker only. isBenignReplyDedup parses the body and requires reason === "already_completed"; anything else — not_delegated, an unparseable body, or a response with no reason at all — is thrown as Reply failed (409). That last one is the back-compat call: an older server sending the unmarked body gets the pre-dedup behaviour (a visible error) rather than the silent drop. Surfacing a conflict the model can act on is recoverable; dropping the user's answer is not.

CONFLICT_UNKNOWN_TASK is deliberately not a third reason. resolve_reply_agent resolves the agent from the task and 403s if it is not this caller's on this topic, so by the time the conflict branch runs the task always exists — a find_by miss in conflict_reason would be unreachable, and it folds into not_delegated (surface it) as the safe default.

Tests pin both directions. Ruby: an answered dispatch replied to twice returns already_completed; cancelled, failed and pending tasks each return not_delegated with an explicit "suppressing this 409 loses the reply" assertion. Making conflict_reason unconditionally return already_completed fails that second test. TypeScript: isBenignReplyDedup over the dedup body, the cancelled body, a legacy no-reason body, and non-JSON.

agents_controller_test 74 runs / 340 assertions / 0 failures; plugin suite 32/32; tsc --noEmit and rubocop clean.

Copilot AI review requested due to automatic review settings July 24, 2026 14:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Copilot AI review requested due to automatic review settings July 24, 2026 14:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

A 409 from /agent/reply has two causes. The dedup one — a sibling session
sharing this agent won the atomic claim and posted — is benign. The other is
a task that left `delegated` WITHOUT being answered (offline-session cancel,
failure, stuck-task recovery mid-turn); the claim fails identically, but
nothing was posted, so swallowing it drops the only copy of the reply.

The server now names the cause: TaskClaimService#conflict_reason returns
already_completed for a done task and not_delegated for everything else, and
reply renders it alongside the error. The client suppresses only on the
explicit already_completed marker — an unmarked body, including one from an
older server, is raised as before dedup existed.
Copilot AI review requested due to automatic review settings July 24, 2026 15:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 96bb8ac0eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


def conflict_reason(agent:, topic:, requested_task_id:)
task = Task.find_by(id: requested_task_id, agent_id: agent.id, topic_id: topic.id)
task&.status == "done" ? CONFLICT_ALREADY_COMPLETED : CONFLICT_NOT_DELEGATED

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Only mark conflicts benign after a reply exists

When the first sibling has claimed a delegated task, claim sets the task to done before comment.save/finalize links the reply; if that worker crashes or the save later fails (for example blank/invalid text, after which the controller restores the task to delegated), a concurrent valid reply can hit this method during the done-without-reply window and receive reason: "already_completed". The plugin now suppresses exactly that reason, so the valid answer is dropped even though no comment was ever created; base the benign reason on an actual linked reply (or otherwise wait for finalization), not status alone. Fresh evidence here is the new status-only already_completed marker.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 03f5a1d. status == "done" was standing in for "the dispatch was answered", and it is not the same claim: #claim flips the task to done, and the controller saves the comment after that returns. Inside that window the row reads completed with nothing posted behind it — and the window can end in a rollback (comment.save false → the controller restores the task to delegated) or never end at all if the worker dies. A concurrent /reply arriving there was told already_completed, so the client suppressed it and dropped the only copy of an answer nobody else had written.

The linked reply comment is the proof, since #finalize is what creates the link (comment.update_column(:task_id, task.id)Task#reply_comment):

return CONFLICT_NOT_DELEGATED unless task&.status == "done"
task.reply_comment.present? ? CONFLICT_ALREADY_COMPLETED : CONFLICT_CLAIMED_WITHOUT_REPLY

A third reason rather than folding it into not_delegated: a claimed-but-unanswered task really is done and really did leave delegated, so reporting not_delegated would put a false statement in the operator-facing half of the payload for the sake of the machine-readable half. The client keys on the one explicit already_completed marker, so a new value is non-benign by construction and needs no client change to be safe — but the test pins it anyway, because "unknown reasons happen to fall through to safe" is not something the next edit should be free to break.

One deliberate consequence: two replies that genuinely race — the second arriving in the microseconds between the winner's claim and its save — now surface a conflict instead of staying quiet, and the caller sees an error for a duplicate. That is the trade this direction has to make. Nothing at 409 time can distinguish "the winner is mid-save" from "the winner died mid-save"; only one of those two mistakes loses a message, and a retry of the spurious one converges (by then the reply is linked, and the second attempt is suppressed as dedup).

Tests both ways: a task set to done with no linked comment — exactly what #claim leaves behind — now answers claimed_without_reply, and reverting the predicate to the status check fails it with already_completed. The plugin side pins that the new reason is not benign.

agents_controller_test 75 runs / 344 assertions / 0 failures; plugin node --test 33/33; rubocop clean on both changed files.

conflict_reason read `status == "done"` as "the dispatch was answered", but
#claim flips the task to done BEFORE the controller saves the comment. Inside
that window the row says completed with nothing posted behind it, and the
window can end in a rollback (invalid text restores the task to `delegated`)
or never end at all (the worker dies). A concurrent /reply arriving there was
told already_completed and the client dropped it — the one copy of an answer
nobody else had written.

The linked reply comment is what #finalize creates, so its presence is the
only proof an answer exists. already_completed now requires it; a claimed but
unanswered task reports claimed_without_reply, which the client does not
suppress. Reasons stay diagnostic — the client suppresses on the one explicit
marker, so the new value needs no client change to be safe.
Copilot AI review requested due to automatic review settings July 24, 2026 15:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03f5a1da02

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


# `reply_comment` is what #finalize links (comment.task_id = task.id), so
# its presence is the only proof the dispatch was actually answered.
task.reply_comment.present? ? CONFLICT_ALREADY_COMPLETED : CONFLICT_CLAIMED_WITHOUT_REPLY

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for finalization before hard-failing duplicate replies

When two same-directory Claude sessions reply at nearly the same time, the loser can reach this check after the winner's claim has committed status = done but before finalize has linked the reply comment. In that normal sibling-race window reply_comment.present? is still false, so the server returns claimed_without_reply; the client then throws instead of treating the race as benign, leaving the original 409-as-user-visible-error behavior timing-dependent even though the winning reply is about to land. Consider waiting/reloading for finalization (or otherwise making claim+link atomic) before deciding this is a hard conflict.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 97e3bda. You are right that the previous round made the loser's outcome timing-dependent, and the direction of that dependence is the bad one: the sibling race is the common case, and it lands in the done-without-reply window every single time. #claim commits done, then the controller runs comment.save — validations, callbacks, broadcasts — and only then does #finalize reach comment.update_column(:task_id, ...). Every winning claim passes through claimed-without-reply on its way to answered, so deciding on the first read was reporting "your answer may be the only copy" for what is almost always a duplicate that is milliseconds from landing.

The predicate is unchanged — only an actually linked comment is ever called benign, and nothing is inferred from status or elapsed time. What changed is that finalization gets a bounded chance to produce the proof before the answer is fixed:

FINALIZE_GRACE_SECONDS = 0.5
FINALIZE_GRACE_INTERVAL = 0.05

deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + FINALIZE_GRACE_SECONDS
loop do
  return true if reply_linked?(task)
  return false if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline

  sleep FINALIZE_GRACE_INTERVAL
end

Monotonic clock so a wall-clock adjustment cannot stretch or collapse the deadline mid-wait. The reads are uncached — that one is load-bearing rather than defensive: ActiveRecord::QueryCache wraps the request, so the identical SELECT 1 FROM comments WHERE task_id = ? would be served from the first (empty) result for the entire wait and the loop could never observe the winner's commit.

Only the window pays for it. A non-done task returns not_delegated before the wait is reached, and a reply that was already linked when the request arrived answers on the first read — both pinned by tests that assert the elapsed time stays under the grace period, so a later edit cannot quietly put the common paths behind a sleep.

I did not take the make-claim-and-link-atomic option, though it is the tighter one on paper. Doing it means holding the SELECT FOR UPDATE across comment.save, which is exactly what the current shape avoids: the long comment above #claim exists because that save fires after_update_commit work — TriggerLoopCheckJob, the stop-button broadcast — and running it inside the claim transaction puts those side effects before the commit they depend on. update_all is there specifically to keep them out. Moving the claim after the save instead reopens the duplicate-comment race the claim was written to close. Waiting buys the same observable behaviour for the loser (it blocks either way, on a lock or on a poll) without moving side effects inside a transaction.

What survives past the deadline is the case you and I both want surfaced: a claim whose worker died mid-save. It reports claimed_without_reply 500ms later than before, and nothing suppresses it.

Four tests in a new task_claim_service_test.rb, and reverting the wait to a single read fails two of them: a link that lands between the first and second read now reads as already_completed (with an assertion that it re-read at all, so a first-read decision fails even if it guesses right), and the never-finalized claim asserts both that the window was actually given and that it is bounded.

task_claim_service_test 4 runs / 22 assertions / 0 failures; agents_controller_test 75 runs / 344 assertions / 0 failures; full pre-push suite 2946 runs / 9267 assertions / 0 failures; rubocop clean on both files.

The winning claim commits `done` a whole comment save before #finalize
links the reply, so every ordinary sibling race passes through
claimed-without-reply on its way to answered. Deciding on the first read
made the loser's outcome a function of how fast the winner's INSERT was:
a hard 409 the model has to recover from, or silence, depending on
timing.

conflict_reason now polls for the link for up to 500ms before answering.
The proof is unchanged — only an actually linked comment is ever called
benign — the wait just gives finalization the moment it needs to produce
one. A claim that never finalizes (worker died mid-save) still surfaces,
one deadline later. Reads are uncached: Rails' per-request QueryCache
would otherwise serve the first empty SELECT for the whole wait.

Paths with nothing to wait for skip it entirely: a non-done task answers
not_delegated immediately, and an already-linked reply answers dedup on
the first read.
Copilot AI review requested due to automatic review settings July 24, 2026 15:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants